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METHOD AND DEVICE FOR^V I DEQJASED— QBS ERVAT I ON AND MEASUREMEN T OF 
^ -THE^EATERAL ENVIRONMENT OF A VEHICLE " 

BACKGROUND OF THE INVENTION 
Field of the invention 

[0002] The invention relates to a method and a device for 
video-based observation and measurement of the lateral 
environment around a road vehicle. 

Related Art of the Invention 

[0003] Depending on the application, initial driver 
assistance systems which are commercially obtainable require a 
compromise between the resolution of the scanning and the size 
of the sensed area. For example, the radar sensor system for an 
ACC application must be restricted to several degrees of the 
sensing area in the horizontal direction while for a parking 
assistance function only a short range and resolution can be 
implemented when sensing the environment next to the vehicle 
using ultrasonic sound. Conventional video-based systems provide 
a good compromise between the resolution and the sensing range 
but generally do not supply any direct distance information. 

[0004] Systems which use the vehicle's own movement to obtain 
three-dimensional environment data from two-dimensional depth of 
profile data are presented, on the one hand, in documents US 4 
179 216 Al and US 4 490 038 Al for monitoring railway tunnels 
and fault- free operations on rail routes and, on the other hand, 
in US 5 278 423 Al in connection with the selective recovery of 
pesticides and the sensing of the stock of trees in plantations. 
In all these systems, a three-dimensional image of the 
environment is generated in a data processing unit by arranging 
a set of successive distance profiles in a series. In order to 
be able to determine the distances between the individual, 
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sequentially recorded profiles, in each case distance sensors 
are arranged on the wheels of the vehicles fitted with the 
systems . 

[0005] A device for sensing the position of an agricultural 
vehicle which is fitted with a scanning sensor is described in 
US 5 809 440 Al . Here, the track of the vehicle is followed by 
means of a global navigation system (GPS) . However, since the 
scanning optical sensor which is used for recording images of 
the plant cover does not supply any distance information, only a 
two-dimensional cartography of the underlying surface can be 
obtained by arranging the sensor information items serially. 

[0006] Driver's assistance systems are increasingly employing 
stereoscopic camera systems by means of which it is possible to 
determine computationally the object width on the basis of the 
known geometric arrangement of the pair of cameras used. In 
stereoscopic image processing, an object is scanned from 
different directions using a pair of cameras. The distance 
between the stereoscopic camera system and the object is then 
determined taking into account the camera positions and 
orientations as well as the camera parameters based on the 
principle of triangulation. 

[0007] DE 199 26 559 Al discloses a method and a device for 
detecting objects in the environment of a road vehicle. In the 
method which is presented, the distance from a moving vehicle is 
calculated by evaluating stereoscopic image pairs and properties 
of the detected object are determined. In particular, a 
distance -based segmentation of images is carried out by means of 
stereoscopic image processing. An object detection process is 
then carried out in the segment image areas. 
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[0008] A similar system is known from EP 108 72 57 A2, which 
describes an arrangement for mounting a stereoscopic camera 
system in a vehicle, which system is used for calculating a 
three-dimensional distance distribution for an object under 
consideration. However, when used in road vehicles such 
stereoscopic image processing systems which are known from the 
prior art can be used only to a limited degree since the 
position and orientation of the camera has to be very precise 
and has to be adjusted with a large amount of effort. 
Furthermore, such an arrangement is not often robust enough for 
everyday use and a wide variety of road conditions. 

SUMMARY OF THE INVENTION 
[0009] The invention is based on the object of providing a 
novel method for video-based observation and measurement of the 
lateral environment of a vehicle and a vehicle having a sensor 
system for carrying out the method as claimed in the preambles 
of patent claims 1 and 15, which method has a high level of 
mechanical robustness and can be adjusted without a high degree 
of complexity. 

[00010] The object is achieved according to the invention by 
means of a method and a device having the features of patent 
claims 1 and 15. Advantageous refinements and developments of 
the invention are disclosed in the subclaims. 

[00011] According to the invention, the observation and 
measurement of the lateral environment of a vehicle, mainly in 
order to detect parking spaces, are carried out, on the one 
hand, by means of a camera with which digital images are 
recorded and, on the other hand, by means of a data processing 
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unit which is used to provide the digital images with a time 
stamp and buffer them. 

[00012] In a particularly advantageous fashion, the device 
also comprises a unit for sensing the vehicle's own movement in 
order to select image pairs from the buffered images on the 
basis of this data. A further means is provided for determining 
the position and orientation of the camera which are present at 
the two recording times. By means of an algorithm for 
stereoscopic image processing it is thus possible to generate a 
local 3D depth image on the basis of the image pairing, with the 
position and orientation of the camera at the recording times 
being taken into account within the scope of a synthetic 
stereoscopic geometry. 

[00013] The novel configuration of the subject matter of the 
invention makes it possible, for the first time, to provide a 
system which, when installed in a road vehicle, allows dynamic 
scenarios such as, for example, the lateral 3D geometry with 
respect to the edge of the roadway, to be sensed from the point 
of view of the vehicle which is acting in an actively dynamic 
fashion and as an advantage of which can evaluate during 
parking. In contrast to a paired camera arrangement such as is 
used in conventional stereoscopic camera systems, the monocular 
camera system according to the invention provides, in 
conjunction with the method according to the invention, the 
advantage of requiring significantly less space. Such a system 
constitutes an advantageous solution for integration in the 
vehicle with respect to design aspects and additionally provides 
the advantage of comparatively low costs by dispensing with a 
second camera. The mechanical robustness is comparatively 
greater with monocular camera systems than with stereoscopic 
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cameras. A further advantage which is provided by individual 
camera systems is a significantly reduced degree of complexity 
with respect to mounting and adjustment. 

[00014] In one beneficial embodiment of the invention it is 
conceivable to determine the vehicle's own movement and thus the 
distance covered on the basis of the sensed number of rotations 
of the wheel in order to select images to form image pairs for 
the stereoscopic image processing. Further possible ways of 
sensing the movement of a road vehicle are also provided by 
systems for mobile vehicle navigation which mainly operate with 
GPS (Global Positioning System) and thus provide information 
about the precise position. Vehicle movement dynamics systems 
are also being increasingly used in road vehicles which provide 
a large amount of data on a vehicle's own movement while taking 
into account engine data and information relating to the 
steering system and brakes. 

[00015] In a particularly beneficial fashion it is 
conceivable, in particular at low speeds, that the images to be 
selected to form image pairs do not necessarily need to be two 
directly successive image recordings. For two image recordings 
to form one image pair, the time stamp should ideally be 
selected in such a way that the time difference, after 
conversion in order to take into account the vehicle's own 
speed, corresponds to a permanently predefined distance 
(stereoscopic basis) . However, the distance (stereoscopic basis) 
should preferably be selected such that depending on the current 
speed at the time of two image recordings it lies in a range 
between 0.2m and lm. Since the average speed of a vehicle when a 
road vehicle is parked is approximately 12Km/h, the distance 
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which is covered (stereoscopic basis) of 0.13m between the two 
image recordings is ideal. 

[00016] In a further advantageous refinement of the invention 
it is conceivable that only buffered image pairs during whose 
recording the camera orientation has not changed significantly 
can be used for further processing. A change in the orientation 
of the camera can be brought about by a change in position of 
the vehicle, owing, for example, to greatly changing road 
conditions . A change in the orientation of the camera between 
two recording times leads to a falsification of the result 
during the further processing, similar to a maladjustment of a 
stereoscopic camera system within the scope of the processing of 
stereoscopic images . 

[00017] The subject matter of the invention can be used in a 
particularly advantageous way with the objective of obtaining a 
global 3D depth view from the sequence of accumulated local 3D 
depth views acquired by means of the image pairs. In this 
context, the image data of the individual local 3D depth views 
which can be assigned to the same location points in the 
environment of the vehicle are added to one another and are 
stored in a geometric data structure. The geometric data 
structure is expanded continuously here in the same direction 
with the newly added image areas in accordance with the 
direction of travel of the vehicle. 

[00018] In a further advantageous refinement of the invention, 
the image data of the respective local location points which is 
to be added for a global 3D depth view is subjected to weighting 
before the accumulation process. Owing to this weighting, the 
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noise which has been produced during the accumulation process 
can be reduced by subsequently filtering individual pixels. 

[00019] In one particularly beneficial way, in order to filter 
individual pixels the volume which is represented within the 
scope of the weighting by means of the accumulated 3D depth 
views is divided into individual volume elements and the pixels 
to be added are distributed among the volume elements. All the 
volume elements preferably comprise a predefined uniform volume, 
for example in the form of cubes. 

[00020] It is advantageous to store the individual volume 
elements in a hierarchical data structure, in particular in a 
tree structure. The tree structure is beneficially an octree. In 
which case, with an octree, each node distributes the set of 
stored volumes among eight subtrees (volume elements) by means 
of a 3D key. Each subtree can in turn be subdivided further. 

[00021] The number of pixels within the individual volume 
elements can be beneficially assigned to the volume elements as 
a respective weighting. For this reason, in addition to the data 
of the pixels the weighting of the volume elements is stored in 
the tree structure. 

[00022] It is particularly advantageous to acquire the average 
weighting of all the volume elements and thus determine an 
overall weighting. A threshold value on whose basis it is 
decided whether the pixels contained in the individual volume 
elements are taken into account in the further processing is 
formed. In this context, only those pixels whose assigned volume 
element has a weighting which is equal to or greater than the 
threshold value is taken into account. Only those volume 
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elements which contain pixels are advantageously used to 
determine the overall weighting. 

[00023] The subject matter of the invention can be used 
particularly advantageously with the objective of forming the 
threshold value by multiplying the overall weighting with a 
tuning factor. During the examination of a parking sequence with 
a volume of 8 x 3 x 5 m 3 and a subdivision into 8 cells an 
average weighting of 7.7 for the cells was obtained. With a 
tuning factor of 1.5 the number of pixels per image was thus 
able to be reduced from 46013 to 29792. 

[00024] The camera used within the scope of the invention may 
be a camera which is a component of some other image recording 
system which is already present in the vehicle. 

[00025] The invention will be explained in detail below with 
reference to an exemplary embodiment and a figure. 

[00026] The figure shows, for example, a traffic scenario 
using the device according to the invention for video-based 
observation and measurement of the lateral environment around a 
road vehicle (10) . The measurement of the lateral 3D geometry of 
a traveling vehicle with respect to the edge of the roadway is 
used here to measure a parking space. A parking vehicle (40), a 
crash barrier (50) or some other object (60) which are 
significantly elevated from the surface of the roadway, and thus 
cannot be driven over, may be located at the edge of the 
roadway . 

[00027] For this purpose, a monocular camera (3 0) in 
conjunction with a computer unit is installed in the road 
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vehicle (10) . The camera (30) is configured in such a way that 
the angle between the optical axis and the direction (20) of 
travel is preferably 90°. However, the possible orientation of 
the camera (30) is not restricted to the orientation which is 
perpendicular to the direction (20) of travel but rather may 
have an orientation of an angle of 45° to 135° with respect to 
the direction (20) of travel for areas which are to be sensed. 
The variation in the orientation of the camera may be used 
advantageously in particular for different configuration 
possibilities with respect to the integration into the vehicle 
as a whole. The image sequence which is obtained from the 
movement of the vehicle (10) when recording using the camera 
(30) is provided with the time stamps by means of the computer 
unit and buffered. An algorithm for stereoscopic image 
processing is used to generate from the buffered images a 3D 
image of the parking space taking into account the vehicle's 
(10) own movement. On the basis of the 3D information it is 
possible to decide, taking into account a defined parking 
strategy, whether the sensed 3D space is suitable for parking 
the vehicle, and whether in particular sufficient space for 
parking is available. 

[00028] For a person skilled in the art it is self-explanatory 
that the subject matter of the invention is also suitable for 
other uses in road traffic. For example, the invention is also 
suitable for providing benefits when used as a warning system 
which measures the lateral distance between a vehicle and the 
edge of the roadway and warns the driver when a minimum distance 
is undershot, in a way which is analogous with the exemplary 
embodiment specified above. 
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